Biased rankings: The centrifugal and centripetal force of randomness

We study the ranking of entities, in terms of a metric which is averaged across the entities’ constituting units, such as countries (entities) being ranked in terms of the average height (metric) of their inhabitants (units). We show that when 1) the entities differ in terms of the number of units they contain (e.g. countries with different numbers of inhabitants), and when 2) part of the variation in the metric across units is due to factors that are not specific to the entity (e.g. the height of an inhabitant of a country is not only due to country specific factors), a particular type of bias in the ranking of entities occurs. More specifically, we show that under these conditions, a centripetal force is exerted on entities with many constituting units. In contrast, a centrifugal force is exerted on entities with few constituting units. In other words, large entities (e.g. countries with many inhabitants) are forced towards the middle of the ranking, while small entities are pushed towards the high and low extremes. We illustrate this insight – which follows from basic probability theory – using numerical simulations, and an empirical assessment of real rankings, in the context of rankings of U.S. States. This insight, which we believe has been overlooked until now, has non-trivial consequences for the credibility of rankings.